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11.2.1 Introduction 

Consider 3. discrete _ time random process ^ TTh — 0,1,2,...}. In the very simple case 
where the X m 's are independent, the analysis of this process is relatively straightforward. In 
this case, there is no "memory" in the system, so each X m can be looked at independently 
from previous ones. 

However, for many real-life processes, the independence assumption is not valid. For 
example, if X m is the stock price of a company at time 771 £ {0,1,2,..}, then it is 
reasonable to assume that the X m 's are dependent. Therefore, we need to develop models 
where the value of X m depends on the previous values. In a Markov chain, X rn+ \ depends 
on X m , but given X m , it does not depend on the other previous values Xq, X\ , • • *, 

X m -\ . That is, conditioned on X m , the random variable is independent of the 

random variables Xq, X\, • • \ X m ^\. 

Markov chains are usually used to model the evolution of "states" in probabilistic systems. 
More specifically, consider a system with the set of possible states s = {si,s 2 ,...}- 
Without loss of generality, the states are usually chosen to be 0, 1,2, • • *, or 1, 2, 3, • • *, 
depending on which one is more convenient for a particular problem. If X n — 2 , we say that 
the system is in state 2 at time 71. The idea behind Markov chains is usually summarized as 
follows: "conditioned on the current state, the past and the future states are independent." 

For example, suppose that we are modeling a queue at a bank. The number of people in the 
queue is a non-negative integer. Here, the state of the system can be defined as the number of 
people in the queue. More specifically, if X n shows the number of people in the queue at time 
72 , then X n £ S = {0,1,2,...} The set S is called the state space of the Markov 
chain. Let us now provide a formal definition for discrete-time Markov chains. 

Discrete-Time Markov Chains 


Consider the random process {X n , 71 — 0,1,2, • • •}, where 

Rxi —Sc {0,1,2, • • •} . We say that this process is a Markov chain if 

P^X m J r i — j\X m — 2 , X m —\ — i m —\, * , Xq — {q ) 

— P[X m -\-\ — j\X m — 2 ), 

for all 771 , j, 2 , Iq , i\ , • • • i m -\. If the number of states is finite, e.g., 

S — {0, 1, 2, • • • , 7*}, we call it a finite Markov chain. 

If X n — j, we say that the process is in state j. The numbers P(X m+ i — j\X m — l) 
are called the transition probabilities. We assume that the transition probabilities do not 
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depend on time. That is, P{X m .|_i = j\X m — l) does not depend on m. Thus, we can 
define 

Pij -^(^m+1 3 1 X m 

In particular, we have 

Pij = P(X ! = j\x 0 = i) 

= P(X 2 = j\Xj = i) 

= P(X 3 =j\X 2 = *) = •••• 

In other words, if the process is in state i, it will next make a transition to state j with 
probability Pij. 
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